Generalizability and Dependability of Single-Item and Multiple-Item Direct Behavior Rating Scales for Engagement and Disruptive Behavior

Direct behavior rating (DBR) has been described as a hybrid of systematic direct observation and behavior rating scales. Although single-item (DBR-SIS) and multi-item (DBR-MIS) methods have been advocated, the overwhelming majority of research attention has focused on DBR-SIS. This study employed generalizability theory to compare the dependability of the two DBR methods for two behaviors (Academic Engagement/Motivation and Disruptive Behavior). Two graduate students used both methods to rate 10-min video clips of the classroom behavior of 8 middle school students on three occasions. Generalizability of ratings was examined across raters and occasions, and decision studies were conducted to determine the minimal number of ratings necessary to obtain an acceptable level of dependability. Results favor the DBR-MIS method over the DBR-SIS method for making timely decisions across decisionmaking contexts. Results are discussed in terms of their implications for research